{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 2.1 Building a new employee Q&A robot with an LLM\n",
        "\n",
        "## 🚄 Preface\n",
        "You work at an educational content development company, and with the continuous influx of new employees, frequent Q&A demands have led to significant time and resource costs. You decide to use LLMs technology to build a Q&A robot to improve the accuracy and efficiency of responses.\n",
        "\n",
        "## 🍁 Goals\n",
        "After completing this course, you will be able to:\n",
        "\n",
        "* Call Qwen-Max through its API\n",
        "* Learn the working principles of LLM\n",
        "* Understand the limitations of LLM and their solutions\n",
        "\n",
        "## ⚠️ Environment preparation\n",
        "To ensure a smoother experience with the tutorial, we recommend that you first complete the [**Environment Preparation**](https://edu.aliyun.com/course/3130200/lesson/343310285) chapter in the **Alibaba Cloud Large Language Model Senior Engineer ACP Certification Course** (alibabacloud_acp_learning), and ensure that the required environment for the course is correctly installed.\n",
        "\n",
        "\n",
        "## 💻 1. Calling Qwen-Max via API\n",
        "\n",
        "The most direct way to experience LLM is by interacting with them through a web interface (such as [Qwen-Max](https://tongyi.aliyun.com/qianwen/)). However, as a developer, you often need to integrate LLM capabilities into your own applications. You can use the widely adopted OpenAI Python SDK to call the Qwen-Max LLM. You have already installed the necessary dependencies in `1.0 Computing Environment Setup`. Before executing the following code, confirm that you have switched to the newly created Python environment, such as the `Python (llm_learn)` environment created in this example.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i3/O1CN01ytNXS81RGWTNtmph7_!!6000000002084-2-tps-3448-1596.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01ytNXS81RGWTNtmph7_!!6000000002084-2-tps-3448-1596.png\" width=\"800\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i1/O1CN01B9bNMT27MDFvpBmnc_!!6000000007782-2-tps-1944-448.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01B9bNMT27MDFvpBmnc_!!6000000007782-2-tps-1944-448.png\" width=\"800\">\n",
        "</a>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To invoke Qwen, you need to go to Model Studio, Alibaba Cloud's LLM service platform, and activate [Model Invocation Service](https://bailian.console.aliyun.com/#/model-market) and [Create an API key](https://bailian.console.aliyun.com/?apiKey=1#/api-key).\n",
        "\n",
        "> If the following is displayed at the top of the page, it means that you have not yet activated the Model Studio invocation service. After activating the service, you can invoke the model.\n",
        "> <img src=\"https://img.alicdn.com/imgextra/i4/O1CN017dWRx21XtLkBg4rMZ_!!6000000002981-2-tps-1236-254.png\" width=\"600\">\n",
        "\n",
        "Before using the API, you need to properly handle the security issues of the API key. Writing the API key directly into the code is a bad habit because it is easy to leak the key when sharing the code, and all parts of the API key encoded in plaintext need to be modified after replacing the API key. A safer and more convenient approach is to store the API key in an environment variable.\n",
        "\n",
        "The following code will load your API key from the configuration file and set it as an environment variable. After the code is executed, the first five characters of the API key will be displayed (followed by asterisks), so you can confirm whether the configuration is correct without exposing the complete key."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T08:58:03.667571Z",
          "iopub.status.busy": "2024-11-15T08:58:03.667184Z",
          "iopub.status.idle": "2024-11-15T08:59:44.060371Z",
          "shell.execute_reply": "2024-11-15T08:59:44.059650Z",
          "shell.execute_reply.started": "2024-11-15T08:58:03.667545Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Your configured API Key is: sk-98*****\n"
          ]
        }
      ],
      "source": [
        "# Load the API Key for invoking the Qwen LLM\n",
        "import os\n",
        "from config.load_key import load_key\n",
        "load_key()\n",
        "print(f'Your configured API Key is: {os.environ[\"DASHSCOPE_API_KEY\"][:5]+\"*\"*5}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You need to press “Enter” to confirm the “API-KEY” you entered. After successful input, you will see the message ```“The API Key you configured is: sk-88***** ”```. \n",
        "If you need to change the “API-KEY”, please edit the “KEY.json” file in the parent directory. \n",
        "If you are using VS-CODE, the input box for the API-KEY will appear at the **top** of the window.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's start with a simple conversation. The following code creates an assistant named \"Company Assistant,\" which can answer questions about company operations. You can use the common question \"Choosing a project management tool\" as an example:  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T08:59:46.180449Z",
          "iopub.status.busy": "2024-11-15T08:59:46.179984Z",
          "iopub.status.idle": "2024-11-15T08:59:59.498380Z",
          "shell.execute_reply": "2024-11-15T08:59:59.497544Z",
          "shell.execute_reply.started": "2024-11-15T08:59:46.180423Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Choosing the right project management tools depends on several factors including the size of your team, the complexity of your projects, your budget, and specific needs such as collaboration, task tracking, and reporting. Here are some popular options that cater to a range of requirements:\n",
            "\n",
            "1. **Asana** - Great for teams of all sizes, Asana offers robust features for task management, project tracking, and team collaboration. It supports both agile and traditional project management methodologies.\n",
            "\n",
            "2. **Trello** - Known for its simplicity and visual approach, Trello uses boards, lists, and cards to organize tasks and projects. It’s particularly useful for teams that prefer a more flexible, less structured approach to project management.\n",
            "\n",
            "3. **Jira** - Ideal for software development teams, Jira is a powerful tool for agile project management. It integrates well with other Atlassian products like Confluence and Bitbucket, making it a comprehensive solution for development workflows.\n",
            "\n",
            "4. **Monday.com** - This platform is highly customizable and can be tailored to fit various business needs. It’s known for its user-friendly interface and visual workflow management capabilities.\n",
            "\n",
            "5. **Microsoft Project** - Suitable for large enterprises, Microsoft Project offers advanced planning and resource management features. It integrates seamlessly with other Microsoft Office applications.\n",
            "\n",
            "6. **Basecamp** - Basecamp is a straightforward tool that focuses on communication and file sharing. It’s particularly good for remote teams and long-term projects.\n",
            "\n",
            "7. **Smartsheet** - Combining the familiarity of spreadsheets with project management features, Smartsheet is a versatile tool that can handle a wide range of project types and sizes.\n",
            "\n",
            "8. **Wrike** - Wrike is another strong option for businesses looking for a scalable solution. It supports multiple project management methods and has strong collaboration features.\n",
            "\n",
            "9. **ClickUp** - ClickUp is an all-in-one productivity platform that includes project management, document management, and goal setting. It’s highly customizable and suitable for teams of any size.\n",
            "\n",
            "10. **Zoho Projects** - Part of the Zoho suite of business apps, Zoho Projects is a cost-effective solution that offers comprehensive project management features including time tracking and issue tracking.\n",
            "\n",
            "When selecting a tool, consider:\n",
            "- **Ease of use**: How quickly can your team get up to speed?\n",
            "- **Integration capabilities**: Does it integrate with other tools your team already uses?\n",
            "- **Scalability**: Can it grow with your business?\n",
            "- **Cost**: What is the total cost of ownership, including any additional features or plugins?\n",
            "\n",
            "It might also be beneficial to try out a few different tools with free trials to see which one best fits your team's workflow and preferences.\n"
          ]
        }
      ],
      "source": [
        "from openai import OpenAI\n",
        "import os\n",
        "\n",
        "client = OpenAI(\n",
        "    api_key=os.getenv(\"DASHSCOPE_API_KEY\"),\n",
        "    base_url=\"https://dashscope.aliyuncs.com/compatible-mode/v1\",\n",
        ")\n",
        "\n",
        "def get_qwen_response(prompt):\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"qwen-plus-0919\",\n",
        "        messages=[\n",
        "            # The system message is used to set the role and task of the LLM\n",
        "            {\"role\": \"system\", \"content\": \"You are an assistant responsible for answering questions in a global educational content development company. Your name is Alex, and you're here to help your colleagues with their inquiries.\"},\n",
        "            # The user message is used to input the user's question\n",
        "            {\"role\": \"user\", \"content\": prompt}\n",
        "        ]\n",
        "    )\n",
        "    return response.choices[0].message.content\n",
        "\n",
        "response = get_qwen_response(\"What tools should our company use for project management?\")\n",
        "print(response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you want to implement multi-round conversations (allowing the LLM to reference historical dialogue information for replies), you can refer to [multi-round conversation](https://help.aliyun.com/zh/model-studio/user-guide/text-generation#865b38621dwin).\n",
        "\n",
        "After running the code above, you will notice that it takes some time (about 20 seconds) to see the complete response. This is because, by default, the API waits until the model has generated all the content before returning the result in one go. In practical applications, this waiting period might affect the user experience — imagine a scenario where users are staring at a blank interface for 20 seconds!\n",
        "\n",
        "Fortunately, you can use \"streaming output\" to optimize this issue. With streaming output, the model outputs responses progressively as it generates them, similar to how humans type, allowing users to see partial responses immediately. This significantly enhances the interactive experience. Next, let’s take a look at how to implement streaming output...\n",
        "> 💡 Tip: Streaming output only changes the way content is displayed; the model's reasoning process and the quality of the final answer remain unchanged. You can confidently use this feature to improve your application's user experience.\n",
        "\n",
        "To implement streaming output, simply add the `stream=True` parameter to the previous code and adjust the output method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T09:00:04.752657Z",
          "iopub.status.busy": "2024-11-15T09:00:04.752276Z",
          "iopub.status.idle": "2024-11-15T09:00:18.769987Z",
          "shell.execute_reply": "2024-11-15T09:00:18.769374Z",
          "shell.execute_reply.started": "2024-11-15T09:00:04.752633Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Choosing the right project management tools can significantly enhance efficiency and collaboration within your team. Here are some popular and effective options that might suit your company's needs:\n",
            "\n",
            "1. **Asana** - Ideal for task and project tracking. It offers features like task assignments, deadlines, and progress tracking, which are essential for managing multiple projects and teams.\n",
            "\n",
            "2. **Trello** - Uses boards, lists, and cards to organize tasks and projects visually. It’s great for teams that prefer a more visual approach to project management.\n",
            "\n",
            "3. **Jira** - Specifically designed for software development teams, Jira supports agile methodologies and integrates well with other Atlassian tools like Confluence and Bitbucket.\n",
            "\n",
            "4. **Microsoft Project** - A powerful tool for detailed project planning and resource management. It’s suitable for large-scale projects and complex scheduling needs.\n",
            "\n",
            "5. **Monday.com** - Offers a flexible platform that can be customized to fit various project management styles. It’s known for its user-friendly interface and robust reporting features.\n",
            "\n",
            "6. **Basecamp** - Known for its simplicity and ease of use, Basecamp is great for small to medium-sized teams. It includes tools for messaging, document sharing, and scheduling.\n",
            "\n",
            "7. **Slack** - While primarily a communication tool, Slack can integrate with many project management tools, making it a central hub for team communication and collaboration.\n",
            "\n",
            "8. **Smartsheet** - Combines project management with spreadsheet functionality, making it useful for detailed planning and data-driven decision-making.\n",
            "\n",
            "9. **Wrike** - Supports collaborative project management with features for task management, time tracking, and custom workflows.\n",
            "\n",
            "10. **ClickUp** - An all-in-one productivity platform that includes project management, docs, goals, and more. It’s highly customizable and scalable.\n",
            "\n",
            "When choosing a tool, consider the following factors:\n",
            "- **Team size and structure**: Some tools are better suited for small teams, while others can handle larger organizations.\n",
            "- **Project complexity**: More complex projects might require more advanced features.\n",
            "- **Budget**: Costs can vary widely, from free plans to enterprise-level pricing.\n",
            "- **Integration capabilities**: Ensure the tool can integrate with other software and systems your team uses.\n",
            "- **User experience**: The tool should be intuitive and easy to use to ensure adoption by your team.\n",
            "\n",
            "It might also be helpful to try out a few options with free trials to see which one best fits your team’s workflow and preferences."
          ]
        }
      ],
      "source": [
        "def get_qwen_stream_response(user_prompt, system_prompt):\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"qwen-plus-0919\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system_prompt},\n",
        "            {\"role\": \"user\", \"content\": user_prompt}\n",
        "        ],\n",
        "        stream=True\n",
        "    )\n",
        "    for chunk in response:\n",
        "        yield chunk.choices[0].delta.content\n",
        "\n",
        "response = get_qwen_stream_response(user_prompt=\"What tools should our company use for project management?\", system_prompt=\"You are an assistant responsible for answering questions in a global educational content development company. Your name is Alex, and you're here to help your colleagues with their inquiries.\")\n",
        "for chunk in response:\n",
        "    print(chunk, end=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "By querying the Qwen-Max model twice, you might notice some interesting phenomena:\n",
        "1. **Even with identical questions**, the responses may vary slightly each time. This is one of the inherent characteristics of LLMs. Much like humans, LLMs can express similar ideas in different ways.\n",
        "2. **The suggestions provided by the model** tend to focus on commonly used project management tools such as Jira, Trello, or Asana. While LLMs possess broad knowledge, they don't have access to your company's specific context — for example, your current toolchain, team size, or budget constraints.\n",
        "> 💡 You can find information about the project management software currently used in your company in the file docs/Content_Developer_Job_Guide.pdf.\n",
        "\n",
        "These two observations are actually quite fascinating! Why do LLMs behave this way? To understand this, we need to lift the \"mysterious veil\" surrounding LLMs and explore how they think and make decisions. Don’t worry — you'll get a clear and intuitive explanation without any complex technical jargon.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 📚 2. How LLMs work\n",
        "In recent decades, artificial intelligence (AI) has experienced a transformative journey from rudimentary algorithms to the advent of generative AI. This advanced form of AI can produce entirely new content—spanning text, images, audio, and video—by learning from extensive datasets. Such capabilities have significantly propelled the proliferation of AI across myriad applications. Common use cases include intelligent question-answering systems (like Qwen-Max and GPT), creative image generation (such as with Stable Diffusion), and code synthesis tools (like Lingma), making AI more accessible and applicable across various sectors.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i4/O1CN01MYjQ0L26b2SY4V5uX_!!6000000007679-2-tps-2528-998.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i4/O1CN01MYjQ0L26b2SY4V5uX_!!6000000007679-2-tps-2528-998.png\" width=\"600\">\n",
        "</a>\n",
        "\n",
        "Intelligent question answering represents one of the most iconic and extensively utilized applications of LLMs, serving as an exemplary model for understanding how LLMs operate. Below, we'll delve into the workflow of LLMs within Q&A scenarios to offer deeper insights into the underlying technical mechanisms.\n",
        "\n",
        "### 2.1. The Q&A workflow of LLMs\n",
        "Below is an example where the input text “ACP is a very” initiates a query to the LLM. The diagram below shows the complete process from initiating the query to outputting the text.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i4/O1CN01SDFPRE1ZQ9jknWnVa_!!6000000003188-2-tps-2922-1595.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i4/O1CN01SDFPRE1ZQ9jknWnVa_!!6000000003188-2-tps-2922-1595.png\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "The Q&A workflow of LLMs consists of five stages:\n",
        "\n",
        "**Stage 1: Tokenization of input text**\n",
        "\n",
        "A tokenization is the basic unit of text processing in LLMs, typically representing words, phrases, or symbols. We need to break down the sentence “ACP is a very” into smaller units with independent semantic meaning (tokenizations) and assign each tokenization an ID. If necessary, please reference [Tokenization](https://github.com/QwenLM/Qwen/blob/main/tokenization_note.md).\n",
        "\n",
        "<a href=\"https://gw.alicdn.com/imgextra/i1/O1CN019gAS3k1DrhwpIdHl6_!!6000000000270-0-tps-2414-546.jpg\" target=\"_blank\">\n",
        "<img src=\"https://gw.alicdn.com/imgextra/i1/O1CN019gAS3k1DrhwpIdHl6_!!6000000000270-0-tps-2414-546.jpg\" width=\"800\">\n",
        "</a>\n",
        "\n",
        "**Stage 2: Token embedding**\n",
        "\n",
        "Computers can only understand numbers and cannot directly comprehend the meaning of tokenizations. Therefore, tokenizations must be converted into numerical representations (i.e., vectors) so that they can be understood by computers. Token embedding transforms each tokenization into a vector of fixed dimensions.\n",
        "\n",
        "\n",
        "**Stage 3: Inference by the LLM**\n",
        "\n",
        "The LLM learns knowledge from vast amounts of pre-trained data. When we input new content, such as “ACP is a very,” the LLM combines its learned knowledge to make predictions. It calculates the probabilities of all possible tokenizations and generates a set of candidate tokenizations. Finally, the LLM selects one tokenization as the next output based on these calculations.\n",
        "\n",
        "This explains why, when asked about internal project management tools within a company, the model cannot provide suggestions for internal tools, as its predictive ability is based solely on the pre-trained data and it lacks knowledge of information it has not been exposed to. Therefore, when requiring a Q&A bot to answer domain-specific questions, this issue needs to be addressed specifically, which will be further elaborated in Section 3 of this chapter.\n",
        "\n",
        "**Stage 4: Output tokens**\n",
        "\n",
        "Since the LLM randomly selects from the candidate tokenizations based on their probabilities, this leads to the phenomenon that “even if the question is exactly the same, the answers are slightly different each time.” To control the randomness of the generated content, parameters such as temperature and top-p are commonly adjusted.\n",
        "\n",
        "For example, in the figure below, the first set of candidate tokenizations output by the LLM is “informative (50%),” “fun (25%),” “enlightening (20%),” and “boring (5%).” Adjusting the temperature or top-p parameters will influence the LLM's preference in selecting from the candidate tokenizations, such as choosing the highest probability option, “informative.” You can learn more about these two parameters in Section 2.2 of this chapter.\n",
        "\n",
        "| Candidate Token | Probability |\n",
        "|-----------------|-------------|\n",
        "| informative     | 50%         |\n",
        "| fun             | 25%         |\n",
        "| enlightening    | 20%         |\n",
        "| boring          | 5%          |\n",
        "\n",
        "Specifically, “informative” will continue to be fed back into the LLM to generate subsequent candidate tokenizations. This process is called auto-regressive model, utilizing both the input text and previously generated text. The LLM uses this method to sequentially generate candidate tokenizations.\n",
        "\n",
        "**Stage 5: Output text**\n",
        "\n",
        "The processes of Stage 3 and Stage 4 are repeated until a special tokenization (such as <EOS>, end of sentence) is output or the output length reaches a threshold, concluding the Q&A session. The LLM then outputs all generated content. Of course, you can utilize the streaming output capability of the LLM, which predicts and immediately returns some tokenizations. In this example, the final output would be “ACP is a very informative course.”"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### 2.2 Parameters affecting the randomness of content generation in LLMs\n",
        "\n",
        "Assume a question-and-answer scenario where the user asks: \"What can you learn in the LLMs ACP course?\" To simulate the content generation process of an LLM, we have preset a candidate token set consisting of the following tokens: \"RAG\", \"prompt\", \"model\", \"writing\", and \"drawing\". The LLM will select one from these five candidate tokens as the output (next-token), as shown below.\n",
        "> User question: What can you learn in the ACP course?<br><br>\n",
        "> LLM response: RAG <br>\n",
        "\n",
        "In this process, two important parameters influence the LLM's output: temperature and top_p. These parameters control the randomness and diversity of the content generated by the LLM. Below, we introduce how these two parameters work and how to use them.\n",
        "\n",
        "#### 2.2.1 Temperature: Adjusting the probability distribution of the candidate token set\n",
        "\n",
        "Before generating the next word (next-token), the LLM first calculates an initial probability distribution for the candidate tokens. This distribution represents the probability of each candidate token being selected as the next-token. Temperature acts as a regulator that alters the probability distribution of the candidate tokens, influencing the content generation of the LLM. By adjusting this parameter, you can flexibly control the diversity and creativity of the generated text.\n",
        "\n",
        "To better understand, the figure below illustrates the impact of different temperature values on the probability distribution of the candidate tokens. The plotting code is located in the /resources/2_1 directory.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i2/O1CN01mNFIad1C1fOCzdTol_!!6000000000021-2-tps-3080-1188.png\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i2/O1CN01mNFIad1C1fOCzdTol_!!6000000000021-2-tps-3080-1188.png\" width=\"1000\">\n",
        "</a>\n",
        "\n",
        "The low, medium, and high temperatures in the figure are based on the range [0, 2) of the Qwen-Plus model.\n",
        "\n",
        "As shown in the figure, as the temperature increases from low to high (0.1 -> 0.7 -> 1.2), the probability distribution becomes flatter. The probability of the candidate token \"RAG\" decreases from 0.8 -> 0.6 -> 0.3. Although it remains the most probable token, its probability is now closer to that of other candidate tokens. Consequently, the final output transitions from relatively fixed to increasingly diverse.\n",
        "\n",
        "For different use cases, you can refer to the following recommendations for setting the temperature parameter:\n",
        "- Clear answers (e.g., generating code): Lower the temperature.\n",
        "- Creative variety (e.g., ad copy): Increase the temperature.\n",
        "- No special requirements: Use the default temperature (usually within the medium range).\n",
        "\n",
        "Note that when temperature = 0, although randomness is minimized, it does not guarantee identical outputs every time. For a deeper understanding, you can refer to [the underlying algorithm implementation of temperature](https://github.com/huggingface/transformers/blob/v4.49.0/src/transformers/generation/logits_process.py#L226).\n",
        "\n",
        "Next, let’s experience the effect of temperature. By adjusting the temperature value, ask the same question 10 times and observe the fluctuations in the responses.\n",
        "> The example code for temperature is similar to the upcoming explanation of top_p, so it has been encapsulated for subsequent use."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "ExecutionIndicator": {
          "show": true
        },
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:39.470715Z",
          "iopub.status.busy": "2024-11-15T12:06:39.470351Z",
          "iopub.status.idle": "2024-11-15T12:06:44.434877Z",
          "shell.execute_reply": "2024-11-15T12:06:44.434229Z",
          "shell.execute_reply.started": "2024-11-15T12:06:39.470689Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Output 1: a steed.\n",
            "Output 2: a steed.\n",
            "Output 3: a steed or an equine.\n",
            "Output 4: a steed or an equine.\n",
            "Output 5: a steed or an equine.\n",
            "Output 6: a steed.\n",
            "Output 7: a steed or an equine.\n",
            "Output 8: a steed.\n",
            "Output 9: a steed.\n",
            "Output 10: a steed or an equine.\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "def get_qwen_stream_response(user_prompt, system_prompt, temperature, top_p):\n",
        "    response = client.chat.completions.create(\n",
        "        model=\"qwen-max\",\n",
        "        messages=[\n",
        "            {\"role\": \"system\", \"content\": system_prompt},\n",
        "            {\"role\": \"user\", \"content\": user_prompt}\n",
        "        ],\n",
        "        temperature=temperature,\n",
        "        top_p=top_p,\n",
        "        stream=True\n",
        "    )\n",
        "\n",
        "    for chunk in response:\n",
        "        yield chunk.choices[0].delta.content\n",
        "\n",
        "\n",
        "# Prints streaming output from Qwen-Plus with optional parameters\n",
        "def print_qwen_stream_response(\n",
        "    user_prompt,\n",
        "    system_prompt,\n",
        "    temperature=0.7,\n",
        "    top_p=0.8,\n",
        "    iterations=10\n",
        "):\n",
        "    for i in range(iterations):\n",
        "        print(f\"Output {i + 1}: \", end=\"\")\n",
        "        # Delay to avoid hitting API rate limits\n",
        "        time.sleep(0.5)\n",
        "        response = get_qwen_stream_response(user_prompt, system_prompt, temperature, top_p)\n",
        "        output_content = ''\n",
        "        for chunk in response:\n",
        "            if chunk:  # Avoid NoneType errors\n",
        "                output_content += chunk\n",
        "        print(output_content)\n",
        "\n",
        "\n",
        "# Qwen-Plus Model Info:\n",
        "# - Temperature range: [0, 2), default: 0.7\n",
        "# - Set temperature=0 for deterministic output\n",
        "print_qwen_stream_response(\n",
        "    user_prompt=\"A horse can also be referred to as\",\n",
        "    system_prompt=\"Continue this sentence with a phrase of five words or fewer.\",\n",
        "    temperature=0\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:45.417531Z",
          "iopub.status.busy": "2024-11-15T12:06:45.417146Z",
          "iopub.status.idle": "2024-11-15T12:06:50.629069Z",
          "shell.execute_reply": "2024-11-15T12:06:50.628320Z",
          "shell.execute_reply.started": "2024-11-15T12:06:45.417506Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Output 1: an equine.\n",
            "Output 2: an equine.\n",
            "Output 3: a steed, mount, or equine.\n",
            "Output 4: a steed, gelding, mare, colt.\n",
            "Output 5: a steed, a nag.\n",
            "Output 6: a steed, a nag, gelding.\n",
            "Output 7: a steed, an equine.\n",
            "Output 8: an equine.\n",
            "Output 9: a stallion or mare.\n",
            "Output 10: an equine animal.\n"
          ]
        }
      ],
      "source": [
        "# Set temperature=1.9\n",
        "print_qwen_stream_response(user_prompt=\"A horse can also be referred to as\", system_prompt=\"Continue this sentence with a phrase of five words or fewer.\", temperature=1.9)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "jupyter": {
          "outputs_hidden": true
        },
        "tags": []
      },
      "source": [
        "It can be clearly observed from the experiment that the higher the temperature value, the more varied and diverse the content generated by the model becomes.\n",
        "\n",
        "#### 2.2.2 top_p: Control the sampling range of the candidate token set\n",
        "\n",
        "top_p is a filtering mechanism used to select a \"small subset\" meeting specific conditions from the candidate token set. The specific method is as follows: sort by probability from high to low, and select tokens whose cumulative probability reaches the set threshold to form a new candidate set, thereby narrowing down the selection range.\n",
        "\n",
        "The figure below shows the sampling effect of different top_p values on the candidate token set.\n",
        "\n",
        "<a href=\"https://img.alicdn.com/imgextra/i1/O1CN01xmkonv21sNL6VtQpi_!!6000000007040-0-tps-2732-1282.jpg\" target=\"_blank\">\n",
        "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01xmkonv21sNL6VtQpi_!!6000000007040-0-tps-2732-1282.jpg\" width=\"700\">\n",
        "</a>\n",
        "\n",
        "In the illustration, the blue part represents tokens whose cumulative probability reaches the top_p threshold (e.g., 0.5 or 0.9), forming a new candidate set; the gray part represents tokens that are not selected.\n",
        "\n",
        "When top_p=0.5, the model prioritizes selecting the highest-probability token, i.e., \"RAG\"; when top_p=0.9, the model randomly selects one among \"RAG,\" \"Prompt,\" and \"Model\" to generate output.\n",
        "\n",
        "\n",
        "From this, it can be seen that the impact of the top_p value on the content generated by LLMs can be summarized as follows:\n",
        "- Larger value: Wider candidate range, more diverse content, suitable for creative writing, poetry generation, and other scenarios.\n",
        "- Smaller value: Narrower candidate range, more stable output, suitable for news drafts, code generation, and other scenarios requiring clear answers.\n",
        "- Extremely small value (e.g., 0.0001): Theoretically, the model only selects the highest-probability token, resulting in very stable output. However, in practice, due to factors such as distributed systems and additional adjustments to model outputs, slight randomness may still be introduced, making it impossible to guarantee completely consistent output every time.\n",
        "\n",
        "\n",
        "Below, experience the effect of top_p. By adjusting the top_p value, ask the same question 10 times and observe the fluctuations in the response content."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:52.894507Z",
          "iopub.status.busy": "2024-11-15T12:06:52.894139Z",
          "iopub.status.idle": "2024-11-15T12:06:58.171739Z",
          "shell.execute_reply": "2024-11-15T12:06:58.171137Z",
          "shell.execute_reply.started": "2024-11-15T12:06:52.894481Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Output 1: GamerBrain Pro\n",
            "Output 2: GamerBrain Pro\n",
            "Output 3: GamerBrain Pro\n",
            "Output 4: GamerBrain Pro\n",
            "Output 5: GamerBrain Pro\n",
            "Output 6: GamerBrain Pro\n",
            "Output 7: GamerBrain Pro\n",
            "Output 8: GamerBrain Pro\n",
            "Output 9: GamerBrain Pro\n",
            "Output 10: GamerBrain Pro\n"
          ]
        }
      ],
      "source": [
        "# Qwen-Plus model: The value range of top_p is (0,1], with a default value of 0.8.\n",
        "# Set top_p=0.001\n",
        "print_qwen_stream_response(\n",
        "    user_prompt=\"Name an intelligent gaming smartphone, it could be\",\n",
        "    system_prompt=\"Suggest a short and catchy name for a smart gaming smartphone, using 4 words or fewer.\",\n",
        "    top_p=0.001\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-11-15T12:06:58.394593Z",
          "iopub.status.busy": "2024-11-15T12:06:58.394205Z",
          "iopub.status.idle": "2024-11-15T12:07:03.378361Z",
          "shell.execute_reply": "2024-11-15T12:07:03.377556Z",
          "shell.execute_reply.started": "2024-11-15T12:06:58.394567Z"
        },
        "tags": []
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Output 1: GamerBrain Pro\n",
            "Output 2: GamerPro Smart\n",
            "Output 3: Gamer IQ\n",
            "Output 4: GamerPro AI\n",
            "Output 5: GamerBrain Pro\n",
            "Output 6: GamerBrain Pro\n",
            "Output 7: GamerBrain Pro\n",
            "Output 8: GamerPro AI\n",
            "Output 9: GamerBrain Pro\n",
            "Output 10: GamerPro AI\n"
          ]
        }
      ],
      "source": [
        "# Set top_p=0.9\n",
        "print_qwen_stream_response(\n",
        "    user_prompt=\"Name an intelligent gaming smartphone, it could be\", \n",
        "    system_prompt=\"Suggest a short and catchy name for a smart gaming smartphone, using 4 words or fewer.\", \n",
        "    top_p=0.9\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Based on the experimental results, it can be observed that the higher the top_p value, the greater the randomness in the output of LLMs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### 2.2.3 Summary\n",
        "\n",
        "**Should temperature and top_p be adjusted simultaneously?**\n",
        "\n",
        "To ensure the controllability of the generated content, it is recommended not to adjust top_p and temperature at the same time. Simultaneous adjustment may lead to unpredictable output results. You can prioritize adjusting one parameter, observe its impact on the results, and then fine-tune gradually.\n",
        "\n",
        "><br>**Knowledge Extension: top_k**<br>In the Qwen series models, the parameter top_k also has capabilities similar to top_p. Refer to the [Qwen API Documentation](https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api?spm=a2c4g.11186623.help-menu-2400256.d_3_3_0.68332bdb2Afk2s&scm=20140722.H_2712576._.OR_help-V_1). It is a sampling mechanism that randomly selects one Token from the top k ranked by probability for output. Generally speaking, the larger the top_k, the more diverse the generated content; the smaller the top_k, the more fixed the content. When top_k is set to 1, the model only selects the Token with the highest probability, making the output more stable but also causing a lack of variation and creativity.<br><br>\n",
        ">**Knowledge Extension: seed**<br>In the Qwen series models, the parameter seed also supports controlling the determinism of the generated content. Refer to the [Qwen API Documentation](https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api?spm=a2c4g.11186623.help-menu-2400256.d_3_3_0.68332bdb2Afk2s&scm=20140722.H_2712576._.OR_help-V_1). Passing the same seed value during each model invocation while keeping other parameters unchanged will make the model return the exact same response every time as much as possible, but it cannot guarantee that the results will be completely identical every time.<br><br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "**Why does randomness still exist when controlling the output of LLMs (LLM) by setting temperature, top_p, and seed?**\n",
        "\n",
        "Even if temperature is set to 0, top_p is set to an extremely small value (e.g., 0.0001), and the same seed is used, the generated results for the same question may still be inconsistent. This is because some complex factors may introduce slight randomness, such as the LLMs running in a distributed system or optimization being applied to the model's output.\n",
        "\n",
        "For example:\n",
        "A distributed system is like slicing bread with different machines. Although each machine operates according to the same settings, subtle differences between devices may still result in slightly different slices of bread.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## ⚙️ 3. Enable LLMs to answer private knowledge questions\n",
        "To enable LLMs to answer private knowledge questions, you can choose one of the following two approaches:\n",
        "- **Prompt Injection (No Model Changes)**<br>\n",
        "    Provide the relevant internal knowledge directly within the prompt when making a request.\n",
        "- **Model Customization (Fine-tuning or Training)**<br>\n",
        "    Fine-tune or train a new model to better understand and respond to domain-specific queries.\n",
        "\n",
        "Given the high cost and complexity of model training, for most internal knowledge Q&A use cases, it's often more efficient to include the necessary context directly in the prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Currently, Software Team 1 is using Jira for project management. However, there are plans to transition your team to Microsoft Project by mid-2026. For now, continue using Jira, and you will receive further instructions regarding the transition closer to the planned date. If you have any questions or need assistance with Jira, feel free to reach out!"
          ]
        }
      ],
      "source": [
        "# User's question\n",
        "user_question = \"I'm part of Software Team 1. What project management tool should we be using?\"\n",
        "\n",
        "# Internal knowledge about company tools\n",
        "knowledge = \"\"\"The company provides two main project management tools:\n",
        "  \n",
        "  1. **Jira**: Best suited for software development teams practicing agile methodologies like Scrum and Kanban. It offers robust features such as issue tracking, sprint planning, and time logging.\n",
        "  \n",
        "  2. **Microsoft Project**: Designed for enterprise-level or complex projects requiring detailed planning, resource allocation, and budget control. This is ideal for teams needing strict oversight over timelines and costs.\n",
        "  \n",
        "  Recommendation: While Microsoft Project is the company-standard solution and fully licensed, Software Teams 1, 3, and 4 are currently using Jira. There are plans to transition these teams to Microsoft Project by mid-2026.\n",
        "\"\"\"\n",
        "\n",
        "# Use a named assistant persona (for clarity and branding)\n",
        "response = get_qwen_stream_response(\n",
        "    user_prompt=user_question,\n",
        "    system_prompt=\"You are Alex, an internal support assistant at an educational content development company. Use the provided context to answer the user’s question clearly and accurately.\" + knowledge,\n",
        "    temperature=0.7,\n",
        "    top_p=0.8\n",
        ")\n",
        "\n",
        "for chunk in response:\n",
        "    print(chunk, end=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When you provide reference information directly within a prompt, LLMs can answer questions involving private or internal knowledge. However, this approach has notable limitations. Prompts have maximum length constraints, and including large amounts of background data can lead to overly long inputs. This not only reduces model efficiency but may also hit token limits, preventing the model from generating a response altogether.\n",
        "\n",
        "To address these limitations, you can implement a system that automatically retrieves relevant internal knowledge when a user asks a question. The retrieved document snippets are then combined with the user’s query and fed into the LLM to generate a final answer. This way, you avoid flooding the prompt with excessive background data. This approach is commonly referred to as **Retrieval-Augmented Generation (RAG)**.\n",
        "\n",
        "A typical RAG pipeline consists of two main stages:\n",
        "\n",
        "### 3.1 Indexing phase\n",
        "<img src=\"https://img.alicdn.com/imgextra/i3/O1CN01aDhfCp1LCVJIuZuUd_!!6000000001263-2-tps-5692-2427.png\" width=\"1000\">\n",
        "\n",
        "During the indexing phase, internal knowledge documents or fragments are transformed into a format suitable for fast and accurate retrieval. This involves splitting the content into chunks, converting each chunk into a numerical representation (called an embedding) using specialized models, and storing these embeddings in a vector database.\n",
        "\n",
        "This vectorization process preserves semantic meaning and enables efficient similarity searches across large knowledge bases. As a result, queries can quickly locate the most relevant documents, significantly boosting both the speed and accuracy of responses.\n",
        "\n",
        "### 3.2 Retrieval and generation phase\n",
        "<img src=\"https://img.alicdn.com/imgextra/i1/O1CN01vbkBXC1HQ0SBrC1Ii_!!6000000000751-2-tps-1776-639.png\" width=\"600\">\n",
        "\n",
        "During the retrieval phase, the system searches the vector database for the most relevant document snippets based on the user’s query. These results are then combined with the original question and passed to the LLM, which generates a final, contextually informed answer. This allows the model to effectively respond to questions involving internal or proprietary knowledge without requiring full access to the underlying documents.\n",
        "\n",
        "In summary, applications based on the RAG structure avoid various issues caused by inputting entire reference documents as background information while extracting the most relevant parts through retrieval, thus improving the accuracy and relevance of the LLM output.\n",
        "\n",
        "## 📝 4. Summary\n",
        "In this section, we learned the following:\n",
        "\n",
        "- **How to use the LLM API**<br>\n",
        "    With hands-on code examples, you've seen how to interact with an LLM API and experienced its capabilities in answering questions and performing Q&A tasks.\n",
        "- **A preliminary understanding of how LLMs work**<br>\n",
        "    You now have a basic understanding of how LLMs interpret questions and generate responses. We also discussed some of their limitations — such as randomness and knowledge boundaries — and how techniques like prompt engineering and RAG can help overcome them.\n",
        "\n",
        "Beyond what we've shown here, LLMs are capable of much more than just answering questions. They can generate content, extract structured information, classify text, and analyze sentiment. Integrating RAG into your LLM applications also extends the range of knowledge they can leverage.\n",
        "\n",
        "In the next section, we’ll walk through how to build your own RAG-based applications.\n",
        "\n",
        "### Further reading\n",
        "- While going through this material, if you'd like to dive deeper into any of the topics, feel free to ask the LLM for additional explanations or learning resources.\n",
        "> For example, Qwen-Max includes an enable_search parameter that allows the model to enhance its answers by incorporating real-time web search results during response generation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "As of my knowledge cutoff in July 2024, the **2024 Paris Olympics** had not yet taken place. Therefore, I cannot provide the exact number of gold medals won by **Team America (United States)** at the Paris Olympics.\n",
            "\n",
            "However, historically, the United States has been one of the top-performing nations in the Olympics and often leads in the overall medal count, including gold medals.\n",
            "\n",
            "For the most accurate and up-to-date results from the **2024 Paris Olympics**, I recommend checking official Olympic websites or trusted sports news platforms like:\n",
            "\n",
            "- [Olympic Official Website](https://www.olympic.org)\n",
            "- [NBC Olympics](https://www.nbcolympics.com)\n",
            "- [ESPN Olympics coverage](https://www.espn.com/olympics)\n",
            "\n",
            "Let me know if you'd like historical performance data for Team USA in past Olympics!\n"
          ]
        }
      ],
      "source": [
        "completion = client.chat.completions.create(\n",
        "    model=\"qwen-plus\",  # This example uses qwen-plus. You can replace it with other model names as needed. Model list: https://bailian.console.alibabacloud.com/?tab=doc#/doc/?type=model\n",
        "    messages=[\n",
        "        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
        "        {\"role\": \"user\", \"content\": \"Number of gold medals won by Team America at the Paris Olympics\"},\n",
        "    ],\n",
        "    extra_body={\"enable_search\": True},\n",
        ")\n",
        "print(completion.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The reasoning model [QwQ](https://help.aliyun.com/zh/model-studio/user-guide/qwq) of Alibaba Cloud has strong reasoning capabilities. The model will first output the thought process and then provide the response content.  \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "==================== Reasoning Process ====================\n",
            "\n",
            "Okay, so I need to figure out which number is larger between 9.9 and 9.11. Hmm, let me think. Both numbers start with 9, so the whole number part is the same. That means I have to look at the decimal parts to decide which is bigger. \n",
            "\n",
            "First, I remember that when comparing decimals, you start from the first digit after the decimal point. The number with the larger digit in the first place is bigger. If those are the same, then you move to the next digit, and so on. \n",
            "\n",
            "So let's break them down. \n",
            "\n",
            "9.9 can be written as 9.90 if I add a trailing zero to make it easier to compare with 9.11. That doesn't change its value because 9.9 is the same as 9.90. \n",
            "\n",
            "Now, comparing 9.90 and 9.11:\n",
            "\n",
            "Looking at the tenths place (the first digit after the decimal), 9.90 has a 9, and 9.11 has a 1. Since 9 is greater than 1, that means 9.90 is larger than 9.11, right? \n",
            "\n",
            "Wait a second, let me double-check. Maybe I got confused because 9.11 has two digits after the decimal, but 9.9 only has one. But when comparing, adding a zero to make the same number of decimal places helps. \n",
            "\n",
            "Alternatively, maybe I can convert them to fractions to see. \n",
            "\n",
            "9.9 is the same as 99/10, and 9.11 is 911/100. To compare these fractions, I can get a common denominator. The denominators are 10 and 100, so the common denominator is 100. \n",
            "\n",
            "Converting 99/10 to hundredths: 99/10 = (99*10)/(10*10) = 990/100. \n",
            "\n",
            "So now, comparing 990/100 and 911/100. Since 990 is more than 911, 990/100 is bigger. \n",
            "\n",
            "Therefore, 9.9 is larger than 9.11. \n",
            "\n",
            "Alternatively, maybe I can think of money. If these were dollars, 9.9 would be $9.90 and 9.11 would be $9.11. Obviously, $9.90 is more than $9.11. \n",
            "\n",
            "Hmm, that seems straightforward, but sometimes people get confused because 11 is more than 9 when you just look at the numbers after the decimal. But in this case, since the first digit after the decimal is more important, the 9 in the tenths place of 9.9 makes it bigger than the 1 in the tenths place of 9.11. \n",
            "\n",
            "Let me think if there's another way to see this. Maybe subtract them? \n",
            "\n",
            "9.9 minus 9.11. Let's do the subtraction:\n",
            "\n",
            "9.90\n",
            "-9.11\n",
            "------\n",
            "0.79\n",
            "\n",
            "Since the result is positive (0.79), that means 9.9 is larger by 0.79. \n",
            "\n",
            "Alternatively, 9.11 minus 9.9 would be negative, which also confirms that 9.9 is bigger. \n",
            "\n",
            "Another approach: line up the decimals:\n",
            "\n",
            "9.9\n",
            "9.11\n",
            "\n",
            "But to compare, align the decimal points:\n",
            "\n",
            "9.90\n",
            "9.11\n",
            "\n",
            "Now, starting from the left, both have 9 in the units place. Then tenths place: 9 vs. 1. Since 9 is greater than 1, the first number is larger. \n",
            "\n",
            "I think that's pretty much all the ways I can think of to compare them, and all methods point to 9.9 being larger than 9.11. \n",
            "\n",
            "Wait, just to ensure I didn't make a mistake in the fraction conversion. 9.9 is 99 tenths, which is 990 hundredths. 9.11 is 911 hundredths. Since 990 > 911, that's correct. \n",
            "\n",
            "Yeah, that's solid. I don't think I made a mistake here. So the answer is 9.9 is larger than 9.11.\n",
            "\n",
            "**Final Answer**\n",
            "The larger number is \\boxed{9.9}.\n",
            "==================== Complete Response ====================\n",
            "\n",
            "To determine which number is larger between 9.9 and 9.11, we start by comparing the digits after the decimal point. \n",
            "\n",
            "First, we can write 9.9 as 9.90 to align the decimal places with 9.11. \n",
            "\n",
            "Comparing the tenths place (the first digit after the decimal):\n",
            "- 9.90 has a 9 in the tenths place.\n",
            "- 9.11 has a 1 in the tenths place.\n",
            "\n",
            "Since 9 is greater than 1, 9.90 is larger than 9.11. \n",
            "\n",
            "To confirm, we can convert the numbers to fractions:\n",
            "- 9.9 is equivalent to \\(\\frac{99}{10}\\) or \\(\\frac{990}{100}\\).\n",
            "- 9.11 is equivalent to \\(\\frac{911}{100}\\).\n",
            "\n",
            "Since \\(\\frac{990}{100}\\) is greater than \\(\\frac{911}{100}\\), 9.9 is larger. \n",
            "\n",
            "Subtracting the two numbers:\n",
            "- \\(9.90 - 9.11 = 0.79\\), which is positive, confirming 9.9 is larger.\n",
            "\n",
            "Thus, the larger number is \\(\\boxed{9.9}\\)."
          ]
        }
      ],
      "source": [
        "reasoning_content = \"\"  # Define the complete reasoning process\n",
        "answer_content = \"\"     # Define the complete response\n",
        "is_answering = False   # Determine whether the reasoning process has ended and the response has started\n",
        "\n",
        "# Create a chat completion request\n",
        "completion = client.chat.completions.create(\n",
        "    model=\"qwq-32b\",  # Here, qwq-32b is used as an example; you can replace it with another model name as needed\n",
        "    messages=[\n",
        "        {\"role\": \"user\", \"content\": \"Which is larger, 9.9 or 9.11?\"}\n",
        "    ],\n",
        "    stream=True,\n",
        "    # Uncomment the following to return token usage in the last chunk\n",
        "    # stream_options={\n",
        "    #     \"include_usage\": True\n",
        "    # }\n",
        ")\n",
        "\n",
        "print(\"\\n\" + \"=\" * 20 + \" Reasoning Process \" + \"=\" * 20 + \"\\n\")\n",
        "\n",
        "for chunk in completion:\n",
        "    # If chunk.choices is empty, print usage\n",
        "    if not chunk.choices:\n",
        "        print(\"\\nUsage:\")\n",
        "        print(chunk.usage)\n",
        "    else:\n",
        "        delta = chunk.choices[0].delta\n",
        "        # Print the reasoning process\n",
        "        if hasattr(delta, 'reasoning_content') and delta.reasoning_content != None:\n",
        "            print(delta.reasoning_content, end='', flush=True)\n",
        "            reasoning_content += delta.reasoning_content\n",
        "        else:\n",
        "            # Start responding\n",
        "            if delta.content != \"\" and is_answering is False:\n",
        "                print(\"\\n\" + \"=\" * 20 + \" Complete Response \" + \"=\" * 20 + \"\\n\")\n",
        "                is_answering = True\n",
        "            # Print the response process\n",
        "            print(delta.content, end='', flush=True)\n",
        "            answer_content += delta.content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "- If you are interested in multi-modal LLMs, you can refer to:\n",
        "\n",
        "    * [Visual Understanding](https://www.alibabacloud.com/help/en/model-studio/vision)\n",
        "    * [Omni-modal](https://www.alibabacloud.com/help/en/model-studio/qwen-omni)\n",
        "    * [AI Video Generation](https://www.alibabacloud.com/help/en/model-studio/use-video-generation)  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 🔥 Quiz\n",
        "\n",
        "### 🔍 Single choice question\n",
        "\n",
        "<details>\n",
        "<summary style=\"cursor: pointer; padding: 12px; border: 1px solid #dee2e6; border-radius: 6px;\">\n",
        "<b>What is the purpose of the following code snippet❓ (Select 1.)</b>\n",
        "<pre style=\"margin: 10px 0;\">\n",
        "os.environ[\"OPENAI_API_KEY\"] = \"your-api-key-here\" \n",
        "</pre>\n",
        "\n",
        "- A. Load the API key from disk  \n",
        "- B. Store the API key in memory  \n",
        "- C. Set the API key as an environment variable  \n",
        "- D. Create a new API key  \n",
        "\n",
        "**[Click to view answer]**\n",
        "</summary>\n",
        "\n",
        "<div style=\"margin-top: 10px; padding: 15px; border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
        "\n",
        "✅ **Reference Answer: C**  \n",
        "📝 **Explanation**:  \n",
        "The code injects the API key into the current runtime environment's memory space using the operating system's environment variable interface.\n",
        "</div>\n",
        "</details>\n",
        "\n",
        "---\n",
        "\n",
        "### 📝 Case analysis question\n",
        "<details>\n",
        "<summary style=\"cursor: pointer; padding: 12px;  border: 1px solid #dee2e6; border-radius: 6px; font-color:#000;\">\n",
        "<b>Alex, while developing a writing assistant, encounters the following two scenarios. How should he solve these problems❓</b> \n",
        "\n",
        "- Scenario 🅰️ Lack of creativity in generated content: Every time he asks the model to write an article about \"the development of artificial intelligence,\" the generated content is very similar.  \n",
        "- Scenario 🅱️ Generated content deviates from the topic: When asking the model to write a technical document, the generated content often includes irrelevant information.  \n",
        "\n",
        "**Question:**  \n",
        "1. Based on the LLMs workflow learned in this section, what might be the causes of these issues in the two scenarios?  \n",
        "2. How should the temperature or top_p parameters be adjusted to address these problems?  \n",
        "\n",
        "**[Click to view answer]**\n",
        "</summary>\n",
        "\n",
        "\n",
        "<div style=\"margin-top: 10px;  padding: 15px;  border: 1px solid #dee2e6; border-radius: 0 0 6px 6px;\">\n",
        "\n",
        "#### 🎯 Solution for Scenario A\n",
        "**🔍 Cause Analysis**  \n",
        "The `temperature` value is too low (e.g., 0.3), causing the model to make single choices, resulting in a lack of diversity in the generated content.\n",
        "\n",
        "**⚙️ Parameter Adjustment**  \n",
        "\n",
        "\n",
        "\n",
        "```python\n",
        "temperature = 0.7~0.9  # Increase creativity\n",
        " top_p = 0.9            # Expand the range of word selection\n",
        "```\n",
        "\n",
        "#### 🎯 Solution for Scenario B\n",
        "**🔍 Cause Analysis**  \n",
        "`temperature` is too high (e.g., 1.2) or `top_p` is too large\n",
        "\n",
        "**⚙️ Parameter Adjustment**  \n",
        "\n",
        "\n",
        "\n",
        "```python\n",
        "temperature = 0.5~0.7  # Reduce randomness\n",
        " top_p = 0.7~0.8        # Focus on high probability words\n",
        "```\n",
        "\n",
        "🌟 **Parameter Tuning Tips**\n",
        "> It is recommended to adjust the parameters by ±0.2 each time and observe the effect changes through A/B testing.\n",
        "> If you need to balance Scenario A and Scenario B, it is recommended to use the combination: `temperature=0.6` + `top_p=0.8`</div>\n",
        "</details>  \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "llm_learn",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.10"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}
